01. Introduction to Data Ingestion

Navigating the Machine Learning Pipeline

This lesson focuses on the practical aspects of handling data in the machine learning pipeline, using Python as the primary toolset.

Key Steps:

• Setting Up Your Environment

  • Use a simple setup like Jupyter Lab or Jupyter Notebook on a laptop.

• Data Ingestion

  • Determine the current data format and its destination.
  • Best practice is to use Pandas for data import unless handling exceptionally large datasets (>100GB).

• Organizing Data

  • Store data files in a project subfolder titled 'Data' for easy access.
  • Prefer CSV format for downloading and organizing data files.

• Using Pandas for Data Handling

  • Pandas is recommended for importing, manipulating, and exporting relational data.
  • Supports various file types like CSV and Excel.
  • Offers robust features and quick resolutions for bugs due to a large user base.

• Alternatives to Pandas

  • Consider libraries like Dask, PySpark, or Polars when dealing with very large data or requiring faster computations.

Understanding how to effectively ingest and manage data using Pandas lays a foundation for success in machine learning projects.